Download File From SFTP

Introduction

This document outlines the process for creating a Cloud Scheduler job to download files to an SFTP server. The job will download specific files according to a scheduled time and frequency.

Creating a Scheduler for SFTP Download

To create a scheduler for this task, follow these steps:

1. Navigate to Cloud Scheduler in the Google Cloud Console.
2. Select CREATE JOB to begin setting up the scheduler.

Preparation Checklist

Ensure the following resources are in place before scheduling the job:

• SFTP server
• Pub/Sub topic
• Google Cloud Storage bucket

Setup Requirements

Create a Cloud Scheduler job with the following parameters:

• Name: Download File From SFTP
• Region: europe-west2 (Select the region closest to your SFTP server for optimal performance)
• Frequency: 9 10 * * 1 (This cron expression schedules the job to run every Monday at 9:00 AM)
• TimeZone: British Summer Time (BST)
• Target Type: Pub/Sub (This will trigger a Pub/Sub topic when the job is executed)
• Cloud Pub/Sub Topic: projects/<projectId>/topics/downloadFileFromSFTP (Replace <projectId> with your actual project ID)

JSON Object for SFTP Download Scheduler

The following JSON object will be used in the Pub/Sub message payload:

{
  "sftpHostAddress": "eu-central-1.sftpcloud.io",
  "username": "90aed4d72ece4e17ae34bf08834ba559",
  "password": "e6BRyesOeEjT5zWJPdTausExz3DsO1",
  "bucketName": "biddirect-2.appspot.com",
  "sourceFolderName": "./Inbound/SubFolder/",
  "sourceFileName": "Digital_INNK_Fleetlist_{{yyyymmdd}}.csv",
  "downloadFolderName": "DownloadSftp",
  "downloadFileName": "Digital_INNK_Fleetlist.csv",
  "emailId": "mailgundev@digitalinnk.com",
  "errorEmailId": "mailgundev@digitalinnk.com",
  "emailSubject": "SFTP File Download"
}

Payload Description

The payload for configuring Cloud Scheduler includes the following parameters:

Sr.No	Payload	Description
1	sftpHostAddress	The address of the SFTP server from which files will be downloaded.
2	username	The username for authentication with the SFTP server.
3	password	The password associated with the username for SFTP server access.
4	bucketName	The Google Cloud Storage bucket name where downloaded files will be stored.
5	sourceFolderName	The folder on the SFTP server containing the files to be downloaded.
6	sourceFileName	Accepts a template for filenames where a specific date in the format YYYYMMDD is inserted into the name. Example: Digital_INNK_Fleetlist_{{yyyymmdd}}.csv if the current date is 27 Aug 2024, then file with name Digital_INNK_Fleetlist_20240827.csv will be downloaded.
7	downloadFolderName	The folder within the Google Cloud Storage bucket where downloaded files will be stored.
8	downloadFileName	The name to be used for the file once it is downloaded to Google Cloud Storage.
9	emailId	The email address to receive notifications upon successful downloads.
10	errorEmailId	The email address to receive notifications if download fails.
11	emailSubject	The subject line for notification emails post-download.

Scheduler Creation Instructions

To set up the scheduler, follow these steps:

1. Define the Scheduler: Configure the name, region, frequency, and timezone.
2. Configure Scheduler: Target the Pub/Sub topic you created.
3. Optional Settings: Add any required message body or authentication details.

Testing Procedure

To test the setup, use a free trial demo account from SFTP Cloud:

1. Visit the website and click Create test SFTP server.
2. Note the temporary host, username, password, and SFTP host details (valid for 1 hour).
3. Update the Pub/Sub payload with these details and save the changes.
4. Upload a file to the SFTP server using this temporary setup.
5. Configure and run the scheduler to download the file.
6. The process will be logged in the Di-Admin under the Process Log section.
   

Tip: Use FileZilla to configure the connection and verify the downloaded file.

Status: Approved
Category: Protected
Authored By: Jeyakumar Arunagiri on Aug 29, 2024

Revisions